DyVSoR: dynamic malware detection based on extracting patterns from value sets of registers

To control the exponential growth of malware files, security analysts pursue dynamic approaches that automatically identify and analyze malicious software samples. Obfuscation and polymorphism employed by malwares make it difficult for signature-based systems to detect sophisticated malware files. The dynamic analysis or run-time behavior provides a better technique to identify the threat. In this paper, a dynamic approach is proposed in order to extract features from binaries. The run-time behavior of the binary files were found and recorded using a homemade tool that provides a controlled environment. The approach based on DyVSoR assumes that the run-time behavior of each binary can be represented by the values of registers. A method to compute the similarity between two binaries based on the value sets of the registers is presented. Hence, the values are traced before and after invoked API calls in each binary and mapped to some vectors. To detect an unknown file, it is enough to compare it with dataset binaries by computing the distance between registers, content of this file and all binaries. This method could detect malicious samples with 96.1% accuracy and 4% false positive rate. The list of execution traces and the dataset are reachable at: http://home.shirazu.ac.ir/˷ sami/malware